Evaluating Classifiers

In a real-world application of supervised learning, we have a training set of examples with labels, and a test set of examples with unknown labels. The whole point is to make predictions for the test examples. However, in research or experimentation we want to measure the performance achieved by a learning algorithm. To do this we use a test set consisting of examples with known labels. We train the classifier on the training set, apply it to the test set, and then measure performance by comparing the predicted labels with the true labels (which were not available to the training algorithm). Sometimes we have a training set and a test set given already. Other times, we just have one database of labeled training examples. In this case, we have to divide the database ourselves into separate training and test subsets. A common rule of thumb is to use 70% of the database for training and 30% for testing. Dividing the database into training and test subsets is usually done randomly, in order to guarantee that both subsets are random samples from the same distribution. It can be reasonable to do stratified sampling, which means to ensure that each class is present in the exact same proportion in the training and test subsets. It is absolutely vital to measure the performance of a classifier on an independent test set. Every training algorithm looks for patterns in the training data, i.e. correlations between the features and the class. Some of the patterns discovered may be spurious, i.e. they are valid in the training data due to randomness in how the training data was selected from the population, but they are not valid, or not as strong, in the whole population. A classifier that relies on these spurious patterns will have higher accuracy on the training examples than it will on the